Skip to content

Conversation

@vishalbollu
Copy link
Contributor

closes #1250


checklist:

  • run make test and make lint
  • test manually (i.e. build/push all images, restart operator, and re-deploy APIs)

@vishalbollu vishalbollu requested a review from deliahu September 26, 2020 03:33
if rollBackErr != nil {
return nil, "", err
}
return nil, "update failed; rollbacked to previous deployment", nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might still be good to return the error, in case the user can do anything about it

return nil, "", err
rollBackErr := rollBack(prevVirtualService)
if rollBackErr != nil {
return nil, "", err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be good to return both errors, and say something along the lines of “update failed” with the error, and “During rollback, another error occurred” with that error


if err := applyK8sResources(api, prevDeployment, prevService, prevVirtualService); err != nil {
go deleteK8sResources(api.Name)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we still delete what may have been created?


return nil, "", err
}
err = operator.AddAPIToAPIGateway(*api.Networking.Endpoint, api.Networking.APIGateway, false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove from api gateway if this fails, like we do for batch APIs?


if prevVirtualService.Labels["specID"] != api.SpecID {
if err := config.AWS.UploadJSONToS3(api, config.Cluster.Bucket, api.Key); err != nil {
return nil, "", errors.Wrap(err, "upload api spec")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(above) Should we remove from API Gateway if adding it failed, like we do in batch?

return err
}

if err := operator.UpdateAPIGatewayK8s(prevVirtualService, prevAPI, false); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the third arg be “true”?

@vishalbollu vishalbollu changed the title Rollback to previous deployment if update fails Rollback to previous deployment if update fails [WIP] Sep 26, 2020
@deliahu deliahu deleted the rollback-when-api-update-fails branch April 19, 2021 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perform a complete rollback if one of the update steps fail

3 participants